Word Co-occurrence Matrix and Context Dependent Class in LSA based Language Model for Speech Recognition

A data sparseness problem for modeling a language often occurs in many language models (LMs). This problem is caused by the insufficiency of training data, which in turn, makes the infrequent words have unreliable probability. Mapping from words into classes gives the infrequent words more confident probability, because they can rely on other more frequent words in the same class. In this research, we investigates a class LM based on a latent semantic analysis (LSA). A word-document matrix is commonly used to represent a collection of text (corpus) in LSA framework. This matrix tells how many times a word occurs in a certain document. In other words, this matrix ignores the word order in the sentence. We propose several word co-occurrence matrices that keep the word order. By applying LSA to these matrices, words in the vocabulary are projected to a continues vector space according to their position in the sentences. To support this matrices, we also define a context dependent class (CDC) LM. Unlike traditional class LM, CDC LM distinguishes classes according to their context in the sentences. Experiments on Wall Street Journal (WSJ) corpus show that the word co-occurrence matrix works 3.62%-12.72 better than worddocument matrix. Furthermore, the CDC improves the performance and achieves better perplexity than the traditional class LM based on LSA. When the model is linearly interpolated with the word-based trigram, it gives improvements about 2.01% for trigram model and 9.47% for fourgram model on relative perplexity against a standard word-based trigram LM. Keywords— Latent semantic analysis, Language model, Word cooccurrence matrix, Context dependent class